Contextual user interface

ABSTRACT

Embodiments of the present invention analyze a context in which a user interacts with a computer interface and automatically optimizes the interface for the context. The controller or control mode the user selects for interaction may define the context, in part. Examples of control modes include gesturing, audio control, use of companion devices, and use of dedicated control devices, such as game controllers and remote controls. The different input devices are designed for different tasks. Nevertheless, a user will frequently attempt to perform a task using a control input that is not adapted for the task. Embodiments of the present invention change the characteristics of the user interface to make it easier for the user to complete an intended task using the input device of the user&#39;s choice.

CROSS-REFERENCE TO RELATED DOCUMENTS

This application claims the benefit of priority to U.S. ProvisionalApplication No. 61/654,638, filed Jun. 1, 2012, and titled “CompanionExperiences”, the entirety of which is hereby incorporated by reference.

BACKGROUND

User interfaces for televisions, game consoles, media players, personalcomputers, and other devices have become increasingly complex. Deviceshave more and more applications and programs through which users canaccess an ever-expanding amount of content. Additionally, users have avariety of mechanisms for interacting with the interfaces. An interfaceadapted for one form of interaction may not be suitable or optimized fora second kind of interaction.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used in isolation as an aid in determining the scope of the claimedsubject matter.

Embodiments of the present invention analyze a context in which a userinteracts with a computer interface and automatically optimizes theinterface for the context. The controller or control mode the userselects for interaction may define the context, in part. Examples ofcontrol modes, include gesturing, audio control, use of companiondevices, and use of dedicated control devices, such as game controllersand remote controls. The different input devices are designed fordifferent tasks. Nevertheless, a user will frequently attempt to performa task using a control input that is not adapted for the task.Embodiments of the present invention change the characteristics of theuser interface to make it easier for the user to complete an intendedtask using the input device of the user's choice.

In addition to adapting the interface to the chosen interactionmechanism, interfaces can be additionally optimized based onenvironmental conditions. Environmental conditions include the locationof a person relative to other people and objects in the room as well asthe content displayed on the one or more interface. For example, aninteractive menu could be presented on part of a screen that is notbeing blocked by an object or person standing between the screen and theuser who is trying to interact with the content on the screen. Thus, ifthe bottom right corner of the screen was blocked by a person, then helpinformation would be presented on a different part of the screen.Similarly, if a user has a phone or other object in their right hand,then a gesture-based menu may be optimized for use with a left hand. Forexample, the menu objects may be moved to the left-hand side of thescreen so that the user does not have to reach across their body tovirtually select the interface objects.

Embodiments of the present invention may use various signals toascertain the present context. Exemplary signals include color imagedata gathered from a camera, 3D image data gathered from a depth cameraor other 3D-capable acquisition device, audio signals, and stateinformation from one or more computing devices and controllers in thevicinity.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described in detail below withreference to the attached drawing figures, wherein:

FIG. 1 is a block diagram of an exemplary computing environment suitablefor implementing embodiments of the invention;

FIG. 2 is a diagram illustrating a variety of communication mediumsbetween game consoles, game services, and companion devices, inaccordance with an embodiment of the present invention;

FIG. 3 is a diagram showing computing environment suitable for using aninteraction context to optimize an user interface, in accordance with anembodiment of the present invention;

FIG. 4 is a diagram showing a user interacting with interface optimizedfor using gestures, in accordance with an embodiment of the presentinvention;

FIG. 5 is a diagram showing a user interacting with a interface that isautomatically optimized for use with a companion device that isrecognized by the game console, in accordance with an embodiment of thepresent invention;

FIG. 6 is a diagram showing a user interacting with a help interfaceoptimized for use with a game controller, in accordance with anembodiment of the present invention;

FIG. 7 is a diagram showing a user interacting with a help interfaceoptimized for use with audio control, in accordance with an embodimentof the present invention;

FIG. 8 is a diagram showing a user interacting with a help interfaceoptimized for use with a companion device, in accordance with anembodiment of the present invention;

FIG. 9 is a diagram showing a user interacting with an interfaceoptimized for left handed gestures, in accordance with an embodiment ofthe present invention;

FIG. 10 is a diagram showing a user interacting with an interfaceoptimized for right handed gestures, in accordance with an embodiment ofthe present invention;

FIG. 11 is a diagram showing a user interacting with an interfaceoptimized as a result of an object handled by the user, in accordancewith an embodiment of the present invention;

FIG. 12 is a diagram illustrating using the identification of an objector clothing within the environment to generate a correspondinginterface, in accordance with an embodiment of the present invention;

FIG. 13 is a flowchart showing a method of presenting an interface thatis optimized for a present context, in accordance with an embodiment ofthe present invention;

FIG. 14 is a flowchart showing a method of presenting an interface thatis optimized for a present context, in accordance with an embodiment ofthe present invention; and

FIG. 15 is a flowchart showing a method of presenting an interface thatis optimized for a present context, in accordance with an embodiment ofthe present invention.

DETAILED DESCRIPTION

The subject matter of embodiments of the invention is described withspecificity herein to meet statutory requirements. However, thedescription itself is not intended to limit the scope of this patent.Rather, the inventors have contemplated that the claimed subject mattermight also be embodied in other ways, to include different steps orcombinations of steps similar to the ones described in this document, inconjunction with other present or future technologies. Moreover,although the terms “step” and/or “block” may be used herein to connotedifferent elements of methods employed, the terms should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

Embodiments of the present invention analyze a context in which a userinteracts to interact with a computer interface and automaticallyoptimizes the interface for the context. User interactions with a userinterface may include giving different degrees of attention to theinterface ranging from ignoring the interface to complete attention. Thecontroller or control mode the user selects for interaction may definethe context, in part. Examples of control modes, include gesturing,audio control, use of companion devices, and use of dedicated controldevices, such as game controllers and remote controls. The differentinput devices are designed for different tasks. Nevertheless, a userwill frequently attempt to perform a task using a control input that isnot adapted for the task. Embodiments of the present invention changethe characteristics of the user interface to make it easier for the userto complete an intended task using the input device of the user'schoice.

In addition to adapting the interface to the chosen interactionmechanism, interfaces can be additionally optimized based onenvironmental conditions. Environmental conditions include the locationof a person relative to other people and objects in the room as well asthe content displayed on the one or more interface. For example, aninteractive menu could be presented on part of a screen that is notbeing blocked by an object or person standing between the screen and theuser who is trying to interact with the content on the screen. Thus, ifthe bottom right corner of the screen was blocked by a person, then helpinformation would be presented on a different part of the screen.Similarly, if a user has a phone or other object in their right hand,then a gesture-based menu may be optimized for use with a left hand. Forexample, the menu objects may be moved to the left-hand side of thescreen so that the user does not have to reach across their body tovirtually select the interface objects.

Embodiments of the present invention may use various signals toascertain the present context. Exemplary signals include color imagedata gathered from a camera, 3D image data gathered from a depth cameraor other 3D-capable acquisition device, audio signals, and stateinformation from one or more computing devices and controllers in thevicinity.

Having briefly described an overview of embodiments of the invention, anexemplary operating environment suitable for use in implementingembodiments of the invention is described below.

Exemplary Operating Environment

Referring to the drawings in general, and initially to FIG. 1 inparticular, an exemplary operating environment for implementingembodiments of the invention is shown and designated generally ascomputing device 100. Computing device 100 is but one example of asuitable computing environment and is not intended to suggest anylimitation as to the scope of use or functionality of the invention.Neither should the computing device 100 be interpreted as having anydependency or requirement relating to any one or combination ofcomponents illustrated.

The invention may be described in the general context of computer codeor machine-useable instructions, including computer-executableinstructions such as program components, being executed by a computer orother machine, such as a personal data assistant or other handhelddevice. Generally, program components, including routines, programs,objects, components, data structures, and the like, refer to code thatperforms particular tasks, or implements particular abstract data types.Embodiments of the invention may be practiced in a variety of systemconfigurations, including handheld devices, consumer electronics,general-purpose computers, specialty computing devices, etc. Embodimentsof the invention may also be practiced in distributed computingenvironments where tasks are performed by remote-processing devices thatare linked through a communications network.

With continued reference to FIG. 1, computing device 100 includes a bus110 that directly or indirectly couples the following devices: memory112, one or more processors 114, one or more presentation components116, input/output (I/O) ports 118, I/O components 120, an illustrativepower supply 122, radio 124, and sensor 126. Bus 110 represents what maybe one or more busses (such as an address bus, data bus, or combinationthereof). Although the various blocks of FIG. 1 are shown with lines forthe sake of clarity, in reality, delineating various components is notso clear, and metaphorically, the lines would more accurately be greyand fuzzy. For example, one may consider a presentation component suchas a display device to be an I/O component 120. Also, processors havememory. The inventors hereof recognize that such is the nature of theart, and reiterate that the diagram of FIG. 1 is merely illustrative ofan exemplary computing device that can be used in connection with one ormore embodiments of the invention. Distinction is not made between suchcategories as “workstation,” “server,” “laptop,” “handheld device,”etc., as all are contemplated within the scope of FIG. 1 and referenceto “computer” or “computing device.”

Computing device 100 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by computing device 100 and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable media may comprise computerstorage media and communication media. Computer storage media includesboth volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules orother data.

Computer storage media includes RAM, ROM, EEPROM, flash memory or othermemory technology, CD-ROM, digital versatile disks (DVD) or otheroptical disk storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices. Computer storage media doesnot comprise a propagated data signal.

Communication media typically embodies computer-readable instructions,data structures, program modules or other data in a modulated datasignal such as a carrier wave or other transport mechanism and includesany information delivery media. The term “modulated data signal” means asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media includes wired media such as awired network or direct-wired connection, and wireless media such asacoustic, RF, infrared and other wireless media. Combinations of any ofthe above should also be included within the scope of computer-readablemedia.

Memory 112 includes computer-storage media in the form of volatileand/or nonvolatile memory. The memory 112 may be removable,nonremovable, or a combination thereof. Exemplary memory includessolid-state memory, hard drives, optical-disc drives, etc. Computingdevice 100 includes one or more processors 114 that read data fromvarious entities such as bus 110, memory 112 or I/O components 120.Presentation component(s) 116 present data indications to a user orother device. Exemplary presentation components 116 include a displaydevice, speaker, printing component, vibrating component, etc. I/O ports118 allow computing device 100 to be logically coupled to other devicesincluding I/O components 120, some of which may be built in.Illustrative I/O components 120 include a microphone, joystick, gamepad, scanner, hard/soft button, touch screen display, etc.

Radio 124 transmits are receives radio communications. The computingdevice 100 may be a wireless terminal adapted to received communicationsand media over various wireless networks. Computing device 100 maycommunicate via wireless protocols, such as code division multipleaccess (“CDMA”), global system for mobiles (“GSM”), or time divisionmultiple access (“TDMA”), as well as others, to communicate with otherdevices (not shown in FIG. 1). The radio communications may be ashort-range connection, a long-range connection, or a combination ofboth a short-range and a long-range wireless telecommunicationsconnection. When we refer to “short” and “long” types of connections, wedo not mean to refer to the spatial relation between two devices.Instead, we are generally referring to short range and long range asdifferent categories, or types, of connections (i.e., a primaryconnection and a secondary connection). A short-range connection mayinclude a Wi-Fi® connection to a device (e.g., mobile hotspot) thatprovides access to a wireless communications network, such as a WLANconnection using the 802.11 protocol. A Bluetooth connection to anothercomputing device is second example of a short-range connection. Along-range connection may include a connection using one or more ofCDMA, GPRS, GSM, TDMA, and 802.16 protocols.

Sensor 126 observes device and environmental characteristics andconditions. Exemplary sensors comprise accelerometers, gyroscopes,GPS/Location, proximity sensors, light sensors, and vibration sensors.

Multi-Device User Experience

Turning now to FIG. 2, a networked operating environment 200 comprisingmultiple computing devices that can provide a companion experience isshown, in accordance with embodiments of the present invention. Theenvironment 200 includes a primary device 210, companion devices 252,254, 256, and 258, a wireless router 250, a base station 260, a network230 and a companion experience server 220. These devices are merelyexemplary and are not intended to be limiting.

The primary device 210 may be a game console, media console, or othersuitable computing device that presents titles. Titles may be games,movies, applications, music, videos, television shows, and other mediacontent. The game console may be coupled to a display, such as atelevision.

The companion devices 252, 254, 256, and 258 are computing devices. Acompanion device, as used in this application, is a personal computingdevice that provides a second display. Examples include laptops, PCs,netbooks, smartphones, e-readers, PDAs, and tablets. A companionexperience allows the user to interact with content related to mediaplaying on a primary device 210 through interfaces on the companiondevice. The companion experience may also allow the user to control themedia presentation through an interface on the companion device.

The companion experience server 220 facilitates companion experiences byproviding companion content, companion applications, registering andauthenticating companion devices, facilitating communications betweenprimary devices and companion devices, and performing other tasks. Thecompanion experience server may be accessed via a wide-area network,such as the Internet.

The companion devices 252, 254, 256, and 258 may communicate directlywith the primary device 210 via Bluetooth or through a wired connection.The companion devices could also communicate over a local wirelessnetwork generated by the wireless router 250. These connections could bedirect or they could be routed through the companion experience server.The companion devices could also communicate with the primary device viaa data service facilitated by base station 260. The base station 260could route communications to the primary device through whatevercommunication to the Internet 230 the primary device is using. The basestation 260 could also be in direct communication with the primarydevice, if the primary device is using the same data service.

Context-Specific Interface Engine

Turning now to FIG. 3, context-specific interface engine 300 withcomponents suitable for detecting an interaction context and optimizinga user interface is shown, according to an embodiment of the presentinvention. The engine 300 includes a context determination component310, a context data store 312, a user data store 314, a personrecognition component 316, an object recognition component 318, a devicerecognition component 320, and an interface activation component 322.Context-specific interface engine 300 is merely an example of onesuitable architecture and is not intended to suggest any limitation asto the scope of the use or functionality of the present invention.Neither should the context-specific interface engine 300 be interpretedas having any dependency or requirement related to any single componentor combination of components illustrated therein. The context-specificinterface engine 300 may reside on a single computing device or bedistributed across multiple devices including multiple devices connectedin a home area network, local area network, or wide area network. Partsof individual components may reside in part on a client and part on aserver or otherwise be distributed across multiple devices.

The context determination component 310 determines the presentinteraction context of the user. The context comprises user actions anduser interactions with devices, other people, and content currentlybeing displayed on one or more devices. The context determinationcomponent 310 may use context templates to identify and label a context.A context template may describe characteristics of the context andrecommend interface characteristics that are optimized for the context.The characteristics of the context are used to identify the presentcontext. The recommended interface characteristics are those that areoptimized or recommended for the context.

The context determination component 310 may be involved in building oroptimizing context templates based on user actions observed while usingthe system. User's responses to optimized interfaces may be tracked andused to change context templates. For example, when a user does notinteract with features added to an optimized interface, those featurescould be removed from the template. If users frequently navigate from afirst interface feature to a second feature on a second interface, thenthe first and second interface features may be combined into a newoptimized interface.

In one embodiment, data from user interactions with an interface may becommunicated to server based machine learning and analysis that combineinformation from multiple users to create and modify templates.Templates could then be associated with similarly situated people andused to recognize interactions that indicate an user intention tointeract with an interface in a particular manner. For example, contexttemplates could be associated with 30-35 year old men that use a gamepad with their right hand to navigate a media selection interface. Theuser characteristic could be observed, or gathered from account data, acorresponding template retrieved, and the interface optimized accordingto the template. Context templates may be stored in the server andaccessed on demand or periodically communicated to a computing device.

For example, a gesture-input context template has gesture input as acharacteristic. Gesture input occurs when a user motions with theirbody, or body parts, in a gesture that is recognized by an input device,such as a depth camera. For example, a user may perform a thumbs-upgesture intending to turn up the television volume. In any case, thecontext template may include context characteristics that include theuser not holding a control device, or the user not appearing to interactwith a control device being held, in addition to making recognizablegestures. In response, the optimized user interface features may includelarger selectable objects. Accordingly, the user interface may be zoomedor selectable objects could be increased in size. In another embodiment,the selection area around objects are increased. Each context hasdifferent characteristics and optimized interface features. Somecontexts may share characteristics with other contexts and optimizedinterface features.

Input mode is just one context feature. For example, there could be anentire class of gesture-based contexts including right-handed gestures,left-handed gestures, occluded gestures, and multi-person gestures. Thecontext determination component 310 may use the input mode as a startingpoint to select a context class. As more information is received, thecontext may be updated and the interface adjusted accordingly. Theinterface may be dynamically updated as confidence in a contextualscenario increases or decreases. In general, a lower confidence in aparticular context may result in display of more generic interface thatis not optimized for a particular context but is adapted for use indifferent contexts.

Use of a dedicated device input is another context class. Dedicateddevices include remote controls and game controllers and tend to be veryspecialized and less flexible. Characteristics that are used todetermine that a dedicated device is being used include the user pickingup a dedicated device and use of the dedicated device. This is incontrast to the device being put away or just relocated. In addition tousing a camera to determine whether the user appears to be utilizing thedevice as a control mechanism, signals received from the dedicateddevice can confirm what is observed by the image data. In this case, theinterface may be adapted or optimized for use with the selecteddedicated device. The context determination component 310 may work withthe device recognition component 320 to identify the particulardedicated device.

Companion interactions is another context class. Companion interactioninvolves the use of a companion device, such as a smart phone or tablet.Companion devices may have a touch screen that allows the user to makefine or granular selections and to view additional content. Companiondevices may also have a hard or soft keyboard that allow the user toinput text. Characteristics of companion device interaction includepicking up a companion device. A companion device may be identified byits appearance and by signal data received by the device. For example,data packets communicated from a companion device during set up of acommunication session may include a URL that identifies the device anddevice characteristics.

In one embodiment, the person recognition component 316 is used toassociate individual devices with people. For example, multiple peoplemay be in the room with multiple devices. The person recognitioncomponent 316 can help determine who is doing what with their device.For example, a first person may be associated with a device that issitting on the table being ignored. In this case, the user interfacewould not want to push relevant content to this person's companiondevice or move to an interface context that is optimized for control bya companion device. On the other hand, if a second person is viewinghis/her companion device, supplemental content could be pushed to thedevice to enhance or augment the second person's viewing experience orability to control an interface if desired.

Embodiments of the present invention attempt to anticipate that the userwill want to use a companion device as part of the interfaceinteraction. For example, a user that is attempting to navigate a webbrowser interface using gestures may pick up a companion device shortlyafter selecting a textbox. Embodiments of the present invention mayautomatically open the soft keyboard on the companion device to allowthe user to type using the companion device. This optimization wouldoccur without the user doing anything else other than picking up thecompanion device or, if they are already holding it, moving it in a waythat indicates they are preparing to use it. The context in this casealso takes into account the state of the browser application. In thiscase, the context includes an active textbox. If the user was notholding the companion device, the soft keyboard would not be opened anda gesture keyboard on the main display could be opened instead.

An audio context is another context class where an interface may beoptimized. For example, it may be difficult to differentiate betweenpeople talking in the room and a person wanting to begin to use audiocommands to control an interface. In one embodiment, the audio interfaceis turned on upon detecting that a user that is directly interactingwith the interface using voice commands. The user's attention to theinterface may be determined by image data that confirms the user isviewing the interface, in contrast to speaking with other people. Thepresence of other people in the room may also help determine thecontext. For example, people may be less likely to speak to themselvesand any spoken words could turn on the audio recognition system when asingle person is in the room. In contrast, with multiple people in theroom it may be undesirable to turn the audio recognition system onwithout a more explicit command from the user.

In addition to optimizing the visual appearance of the interface, anentirely new interface may be presented upon determining a context. Forexample, if a user is interacting with a web browser using gestures andthen picks up a game controller, a contextual determination may be madethat the user wants to play a game rather than interact with the webbrowser using the game controller. In this case, instead of optimizingthe web browser for use with a game controller, a game home page orother gaming-related interface could be presented. In one embodiment,the user is automatically brought to their last-played game.

The context determination component 310 may utilize one or more machinelearning algorithms to recognize contextual situations and associatethem with an appropriate context template. In particular, the contextdetermination component 310 may input various user characteristicsstored in user data store 314 to help identify the context that isappropriate. In this way, a user's past behavior and preferences can beincorporated into the context determination process.

The context data store 312 stores context characteristics and additionalinformation. The context characteristics may be determined or stored ascontext templates. In one embodiment, context usage for a user is storedin the context data store 312. In other words, the most commonly usedcontexts may be ascertained for a particular user and used in the futureto disambiguate a context determination. The context data store 312 mayreside on a single computing device or be distributed across multipledevices including multiple devices connected in a home area network,local area network, or wide area network. Data associated with thecontext data store 312 may reside in part on a client and part on aserver or otherwise be distributed across multiple devices.

The object recognition component 318 uses image data to recognizeinanimate objects in the room. This is in contrast to people which arerecognized using person recognition component 316. Object recognitioncomponent 318 may recognize game objects or other objects of particularinterest to determining a present context. For example, objectrecognition component 318 may recognize a game object such as a tennisracket or wizard's hat. Upon recognizing a tennis racket, a tennis gamemay be immediately opened in a mode where the user can play using theracket. In the wizard hat example, the embodiments of the presentinvention may select the user's favorite game that includes a wizard andselect the character based on the hat. In games with differentcharacters, different prominent pieces of clothing or game paraphernaliamay be used as a cue to automatically open that game or select aparticular character.

In another embodiment, the object recognition component 318 recognizesvarious objects that the user is interacting with and establishes anappropriate context. For example, a user drinking a cola with the user'sright hand may want to make gestures with the left hand and not put thecola down. In this case, the gesture interface may be optimized for usewith the left hand.

A device recognition component 320 recognizes control input devices,including dedicated devices and companion devices. The different devicesare recognized using image data, similar to that used by objectrecognition component 318, but may also utilize signal data receivedfrom the companion devices and dedicated devices. Signal data may beused in combination with camera data to identify a particular devicewithin the environment. For example, several game controllers may lookalike. However, using camera data that shows the user's interaction,such as pushing buttons or moving the joystick, the particularcontroller may be identified by correlating the image and control data.In one embodiment, the person recognition component 316 is utilized toassociate a particular person or user with a particular device. In thisway, players could be taken to their last progress point, or game state,by simply picking up the controller.

With companion devices, login data and other user characteristics mayalso be evaluated to associate the device with a particular person orplayer with an account on a primary device. Additionally, like the gamecontroller, observed user interactions with a device may be synchronizedwith control data received from the device to place the device withinthe environment and with a particular person.

The interface activation component 322 activates the appropriateinterface for a context determined by context determination component310. In one embodiment, the interface activation component 322communicates with the active application, where appropriate, andprovides new interface settings that should be used in the context. Theinterface settings may be drawn from a context template.

Contextual Interfaces

FIGS. 4 and 5 illustrate an embodiment of the invention thatautomatically optimizes an interface based detecting a change in theuser's method of interaction with content. The user's interaction methodis an example of context. Different interfaces may be optimizeddepending on the input mechanism employed by the user to interact withthe interface. For example, an optimized gesture interface, such asprovided by a depth camera, includes large buttons and other largeinterface objects. In contrast, a touch screen, joystick, push buttons,or other controller, that provides more granular control may be able toshow more information and smaller interactive objects.

FIG. 4 depicts a user 405 sitting on their couch interacting with aninterface 415 displayed on display device 400. The input mechanism is agesture interface with a performance area captured by camera 410. The405 user makes gestures with their hand, feet, entire body, or acombination to manipulate the interface objects, such as boxes E, F, G,and H.

FIG. 5 shows a similar scene except that the user 405 has picked up agame controller 510. Embodiments of the present invention are notlimited to a game controller. The user could pick up a companion devicesuch as a tablet or smartphone. As can be seen, the new interface 515provides additional options that are compatible with the game controller510. For example, the user can scroll horizontally by pushing thecontrollers A or B buttons. In addition, the boxes E, F, G, and H aresmaller because they will be easier to select with the controller, whichallows the user to manipulate a selector or cursor with a joystick or upand down arrows.

Embodiments of this invention involve two steps. The first step isdetecting that the user has changed their interface interaction method.For example, when the user switches from gestures to a controller theactive context may be updated. The second step is to update the userinterface to accommodate the new method of input. Several differentdetection methods may be used. For example, a camera may be used toactively track a user and observe what input method they are presentlyusing. In one example, a depth camera may be used to identify and tracka user's input or interaction method. A vision based system may be usedto detect specific markers on the input device to determine whether itis a tablet, smart phone, TV remote, or game controller.

The system will also determine the type of device that the user isholding. An algorithm, for example, a machine learning depth shapedetector, may be used to determine the type of device held by the user.Potentially, a series of unique markers or internal sensors are put in,or on, the device in order for the system to better distinguish thedevice. Other markers could be added to help the system effectivelydistinguish between two identical devices, such as two game controllers.Different devices may generate different gyroscope or accelerometeroutput because of the number and location of the sensors within an inputdevice. This output can form a pattern. A reading of internal sensors,such as a gyroscope, within a device picked up by a user may becorrelated to a library of known patterns and used to determine theexact external input device held by the user.

The detection could also take place by evaluating device identificationinformation within radio signals transmitted by a device. The opening ofa communication session between a controller, or other device, and thecomputing device controlling the user interface may also include deviceidentifying information. For example, as described previously, acompanion device may connect to a game console. Upon this connection,the game console and companion device are able to communicate. Thisconnection may be open even though the user is not actively using thecompanion device. Thus, the depth camera could detect that the user hastransitioned away from a gesture interface when the user picks up acompanion device.

The camera could make a broad determination that the user has picked upa device that conforms with one or more companion devices that have opencommunication sessions with the user. The camera could then correlateinputs received from the device with movements detected. For example,the camera could determine that the user was touching the touch screenon the companion device at the same time touch screen input was receivedthrough the open connection. At this point, a definitive determinationcould be made that the user has transitioned to the specific companiondevice input experience. Upon making the determination, the userinterface could be transitioned to one optimized for interaction withthe companion device.

Regardless of the exact method used to detect the companion device, thenext step is changing the user interface. Once the system has determinedthat the user is engaging with a new input device, the UI is notifiedand updated accordingly to reflect the new input. For example, when auser is watching a movie on a device that supports gesture controls andvoice controls, they may use their voice and hand gestures to navigatethe user interface. If they get bored watching the film and decide topick up the controller to play something instead, the context change isdetected. Once the system detects they have picked up the controller, acontroller-navigable user interface appears, allowing them to launch aset of controller-based games.

In another example, a user is playing a game that is primarilycontrolled with a controller, but has a secondary device, such as atablet, that displays additional information. While interacting with thetitle, the user picks up the tablet, and the title automaticallyswitches to a display that augments the tablet experience, such as amap, or an item management screen. This example shows that the userinterface may transition in a way that is both optimized in terms ofsize or controls available as well as generating a totally differentinterface with different content based on a task that is likely to beperformed when the user picks up the new controller. Thus, the UI may beoptimized based on input device capability and likely tasks performedwith the device. The user's history of controller and device usage maybe analyzed to determine the optimal user interface.

FIGS. 6, 7 and 8 illustrate help interfaces that are optimized tointeract with different inputs. As mentioned, inputs are characteristicsof a context for which interfaces are optimized.

FIG. 6 describes a help interface that is optimized for a gamecontroller 510, in accordance with an embodiment of the presentinvention. As can be seen, a user 405 is attempting to interact withinterface 615 using game controller 510. The active input mechanism maybe ascertained through a combination of image data gathered by camera410 and from signal data received from game controller 510. As can beseen, the interface 615 includes a help message 630 that states “press Afor more information.” “A” describes a label with a button on the gamecontroller 510. The user may access a menu or a different interface bypressing A in the particular context. The overall media presentationshown in interface 615 may not change from context to context. Forexample, if the car shown is a commercial or part of a movie then itsappearance may stay the same from context to context. The help overlayand other interactive features of the interface 615 may change withcontext.

Turning now to FIG. 7, a help interface optimized for audio control isshown, in accordance with an embodiment of the present invention. Thefeatures of FIG. 7 are similar to those described previously withreference to FIG. 6. However, notice that the user 405 is no longerholding a controller. Though not shown, the user may be speakingcommands to indicate that an audible interface should be activated. Theinterface 715 includes a help interface 730 that states “say ‘more’ formore information.” In an optimized audio mode, various audio commandsthat are available to the user may be suggested based on the task theuser appears to be completing.

Turning now to FIG. 8, a help interface or overlay that is optimized foruse with a companion device is shown, in accordance with an embodimentof the present invention. Interface 815 is similar to the interfacedescribed previously with reference to FIGS. 6 and 7 but includes a helpoverlay that states, “Select ‘car’ icon on tablet for more information.”This message refers to an icon presently being displayed on the user'scompanion device 810. This help overlay takes into account not only whattype of device the user is attempting to use to control the interface,but also content presently displayed on the device. Contextualinterfaces may be built by exchanging state information betweenapplications running on the companion device 810 and on the devicegenerating the interface 815.

The camera 410 may be used to determine that the user is actuallyholding the companion device 810 and willing to use it as a controlmechanism before the interface is changed to be optimized for use withthe companion device 810. For example, were the companion device 810 issitting on the table, the context could be different than when held by auser.

Turning now to FIG. 9, a contextual interface that is optimized based ona person's location within a game environment is shown, in accordancewith an embodiment of the present invention. FIG. 9 includes players 924and 922 playing a video game that is displayed on display device 900.Display device 900 may be a television. Each player is associated withtheir own split screen interface. Player 922 is associated with splitscreen interface 912 and player 924 is associated with interface 914.

Camera 910 may be a depth camera, or other device capable of generatingdata that may be used to recognize people and objects. The arrowsindicate that the split screen interfaces 912 and 914 may switch placesif the players 922 and 924 switch places. This transition can take placeautomatically upon determining that the context has changed. In thiscase, the changing context is the changing location of the two players.In this way, the players are always exposed to a direct view of thesplit screen interface with which they are interacting regardless oftheir movements.

In addition to transitioning during game play between differentinterface locations, the initial setup may be optimized before game playby doing person recognition and understanding which person is associatedwith which user file in the game. Though not shown, players 922 and 924could be using game controllers or companion devices to interact withthe video game. In another embodiment, they are using gestures for audiocontrol to play the video game. Regardless, their relative location maybe identified and the interface optimized accordingly.

Turning now to FIG. 10 and FIG. 11, user interfaces optimized forright-handed or left-handed use are shown, in accordance with anembodiment of the present invention. FIG. 10 shows a person 1040watching content shown on display device 1000. The content includes aprimary display 1020 and a supplemental content window 1030. Thesupplemental content window 1030 includes selectable object 1032,selectable object 1034, and selectable object 1036. Icons and buttonsare examples of selectable objects. The camera 1010 is able to capturedata that is used to identify people and objects. Additionally, theinformation may be used to determine a person's movements, such as thoseassociated with gestures. In this case, the interface, which has theselectable buttons on the left, is optimized for use with theleft-handed gestures. As can be seen, the player 940 is raising theirleft hand 1042 to make a gesture.

A user's right or left-handed preference may be ascertained a number ofdifferent ways. In one embodiment, the user's history is analyzed toascertain their right or left-handed preference. User's that are knownto be right or left-handed, or to use their right or left hand mostfrequently when gesturing, will be automatically presented with anoptimized interface for their hand of preference.

In addition to previous use, the current user situation may be evaluatedto determine which hand the user is likely to use. In one embodiment, auser is deemed not likely to use a hand that is being put to a differentuse currently. For example, if the hand is the right hand holding aglass, a phone, or is in a pocket, then the presumption may be made thatthe user wishes to use the left-hand and a left-handed optimizedinterface is presented. The context template associated with right orleft-handedness may look at a number of factors, to determine which handthe user is likely to use.

Turning now to FIG. 11, the same interface shown in FIG. 10 is shown,except that it is now optimized for right-hand use. As can be seen, theuser 1040 has their right-hand 1044 raised and is using it to interactwith the display. In one example, in a right or left-handed interface ispresented in anticipation of a user's gestures interfering with anotherperson. For example, if it appears from the image data that a user wouldhave to place their right-hand in front of another person's face to makea gesture with their right hand, then a left-handed interface may bepresented to encourage the user to use their left-hand. In anotherembodiment, if it appears that the user's right hand would be blockedfrom view of the camera 1010 by another person or object standing infront of their right hand, then a left-handed interface would again bepresented to encourage use of the left-hand and avoid the occlusionproblem created by the interfering object. In this way, the contextualdetermination can attempt to avoid a problem by creating an interfacethat is optimized to avoid problems detected in the environmentalcontext.

Turning now to FIG. 12, the identification of an object or clothingwithin the environment to generate a corresponding interface isillustrated, in accordance with an embodiment of the present invention.In FIG. 12, user 1220 holds up a tennis racket 1222 in front of displaydevice 1200 and camera 1210. The camera 1210 captures image data that isused to recognize the tennis racket 1222 and perhaps the person 1220. Inresponse, a tennis game is automatically shown in interface 1212. Thisprevents the user from needing to navigate through various interfaces toplay a virtual tennis game, which may be the most likely reason for theuser to show up in front of the interface with a tennis racket 1122.Alternatively, the tennis racquet could be used as a cue that the person1220 wants to watch tennis. User preferences and viewing history may beused to disambiguate the context.

In another embodiment, articles of clothing are recognized and used toestablish a context and present an appropriate interface. In oneembodiment, a team jersey is recognized and used to automatically selecta similar color jersey or even the exact same team within a video game.For example, a video game having NFL teams may be started with the userplaying their team indicated on their jersey. In a multiple playersetting, multiple people could be assigned teams corresponding withtheir jerseys. When an exact match is not possible, similar colors maybe assigned automatically to the team they are controlling.

Depending on other contextual factors, such as an ongoing sporting eventinvolving the team identified from the clothing, the presentation ofthat sporting event may automatically be presented on the userinterface. For example, if a football game involving the team indicatedon a person's jersey is presently available, then it may be shown. Ifthe game is ongoing, but is not presently available because of theuser's cable package, for example, a mechanism for viewing the game,such as subscribing to a particular cable package may be presented tothe user and automatically set up. Alternatively, a web page showing agame update may be displayed automatically.

Disambiguating the play video game vs. watch game contexts, may hinge onwhether there is a sporting event available involving the relevant team.Thus, contextual factors may include more than just what is going on inthe present environment, but may include evaluation of various contentthat may be of interest to the user or available to the user at aparticular time. When multiple contexts are available, a user's previousviewing history may be considered to help disambiguate the selection.The user may also be presented with alternative interfaces. For example,the user could be asked whether they want to watch their team play theongoing contest or play a video game.

In another embodiment, clothing associated with various video games maybe used to automatically choose characters or various available games.For example, a person wearing a wizard hat may be automatically assignedthe wizard character as they start a game involving a wizard. In aslight variation, in response to wearing the wizard hat, a gameinvolving a wizard may be opened without selecting a particularcharacter. When multiple games involving a wizard are available, themost recently played game involving a wizard may be selected andpresented.

Methods for Generating Contextual Interfaces

Turning now to FIG. 13, a method of presenting an interface that isoptimized for the present context is shown, in accordance with anembodiment of the present invention. The interface of method 1300 may bea computer interface displayed on a television, monitor, or otherdisplay device. The interface may be generated by game console, mediaplayer, personal computer, or other similar device.

At step 1310, a present context for a user interface is determined byanalyzing image data depicting an environment proximate to the userinterface. In one embodiment, the environment proximate to the userinterface is the volume of space depicted in the image data. In anotherembodiment, the environment extends behind the area captured by theimage data and is demarked by physical boundaries such as those in aroom or a residence such as a house or apartment. Though these otherareas may not be in view of a camera that captures the image data,signals from devices in the environment may be received as well asaudible commands from users that are not in view of a camera. In oneembodiment, the image data is three-dimensional image data, such as thatcaptured by a depth camera. The image data may be described as a depthcloud. The three-dimensional data allows the size, shape, and motion ofobjects to be detected within the data. The depth cloud may be used torecognize gestures made by a user. A gesture may be a hand movement,body movement, facial expression, or other action recognized by agesture interpretation component to control an interface.

At step 1320, a user interface that is optimized through a presentcontext is generated. The user interface is output for display to theuser. Various ways of optimizing a user interface have been describedpreviously. For example, user interface may be optimized for use with aparticular control mode. Within the various control modes, the task theuser is intending to complete through the interface may be anticipatedand the user interface updated accordingly. The intended task is part ofthe context that is derived from evaluating the environment. Userinteractions with objects in the environment may also be used todetermine the context.

Turning now to FIG. 14, a method 1400 of presenting an interface isoptimized to the present context is shown, in accordance with anembodiment of the present invention. At step 1410, a present context fora user interface is determined by analyzing actions of a person that isproximate to the user interface. The actions comprise use of a controlmode with the interface. Exemplary control modes include using gestures,using a companion device, using a dedicated device, and audible control.The present context may be more granular than a particular control mode.For example, within a gesture control mode, the context may determinethat right-handed gestures or left-handed gestures are preferable.

At step 1420, a user interface that is optimized to the present contextis generated. The user interface is output for display to the user.Methods of optimizing the user interface in view of the present contexthave been described previously.

Turning now to FIG. 15, a method 1500 of presenting an interface that isoptimized to the present context is shown, in accordance with anembodiment of the present invention. At step 1510, a present context fora user interface is determined by analyzing image data depicting anenvironment proximate to the user interface. Present contexts and theuse of image data to determine the present context have been describedpreviously with reference to FIG. 13 and elsewhere.

At step 1520, a present user interface that is optimized to the presentcontext is generated. The present user interface is output for displayto the user.

At step 1530, a new context for the user interface is determined byanalyzing image data depicting the environment proximate to the userinterface. The image data depicts changes to the environment that resultin the new context. At step 1540, the user interface is automaticallyoptimized for the new context.

Embodiments of the invention have been described to be illustrativerather than restrictive. It will be understood that certain features andsubcombinations are of utility and may be employed without reference toother features and subcombinations. This is contemplated by and iswithin the scope of the claims.

The invention claimed is:
 1. A method of presenting an interface that isoptimized for a present context, the method comprising: determining apresent context for a user interface by analyzing image data depictingan environment proximate to the user interface; and generating the userinterface that is optimized for the present context.
 2. The method ofclaim 1, wherein the present context comprises a control mode a user isutilizing to interact with the user interface.
 3. The method of claim 2,wherein the method further comprises automatically re-optimizing theuser interface upon determining a new context for the interface.
 4. Themethod of claim 2, wherein the control mode comprises one or more ofgestures with no control device, use of a companion device, and audiblecontrol.
 5. The method of claim 2, wherein the user interface isoptimized for a gesture control mode in which the user does not use acontroller.
 6. The method of claim 1, wherein the present contextcomprises an object that a user is interacting with.
 7. The method ofclaim 1, wherein the user interface is optimized by presenting anoverlay adapted for the present context.
 8. One or more computer-storagemedia having computer-executable instructions embodied thereon, thatwhen executed by a computing device perform a method of presenting aninterface that is optimized for a present context, the methodcomprising: determining a present context for a user interface byanalyzing actions of a person that is proximate to the user interface,the actions comprising use of a control mode with the interface; andgenerating the user interface that is optimized for the present context.9. The media of claim 8, wherein the control mode comprises one or moreof gestures with no control device, use of a companion device, andaudible control.
 10. The media of claim 8, wherein the method furthercomprises automatically re-optimizing the user interface upondetermining a new context for the interface.
 11. The media of claim 8,wherein the actions comprise use of a left hand to make a gesture andthe user interface is optimized for use with left handed gestures. 12.The media of claim 8, wherein the actions comprise interaction with anobject and the user interface is optimized by opening an applicationthat is related to the object.
 13. The media of claim 12, wherein theobject is a piece of clothing worn by the person.
 14. The media of claim8, wherein the user interface is optimized by presenting an overlayadapted for the control mode.
 15. A method of presenting an interfacethat is optimized for a present context, the method comprising:determining a present context for a user interface by analyzing imagedata depicting an environment proximate to the user interface;generating a present user interface that is optimized for the presentcontext; determining a new context for the user interface by analyzingimage data depicting the environment proximate to the user interface;and automatically optimizing the user interface for the new context. 16.The method of claim 15, wherein the new context is disambiguated frommultiple possible contexts using previous user interactions with theuser interface.
 17. The method of claim 15, wherein analyzing theenvironment proximate to the user interface comprises determining acontrol mode utilized by a user to interact with the user interface. 18.The method of claim 17, wherein the control mode comprises gesturesinterpreted from a three-dimensional image data.
 19. The method of claim15, wherein the environment comprises user interactions with an objectand the user interface is optimized by opening an application that isrelated to the object.
 20. The method of claim 19, wherein the object isa game object and the application is a game related to the object.